Skip to content

Tuple Sketches: Deserialization improvement + bug fix - #76

Merged
tisonkun merged 4 commits into
apache:mainfrom
cboumalh:cboumalh-tuple-deserialization-fix
Nov 19, 2026
Merged

tisonkun merged 4 commits into
apache:mainfrom
cboumalh:cboumalh-tuple-deserialization-fix

Conversation

@cboumalh

@cboumalh cboumalh commented Nov 10, 2026 •

Copy link
Copy Markdown

Switches the per-entry deserialization buffer from an allocated nb::bytes (which copied the full remaining serialized blob on every retained entry, producing O(N²) copy work on large sketches) to a zero-copy memoryview over the C++ buffer. Updates the PyObjectSerDe.from_bytes contract from bytes to a buffer-protocol object and adjusts PyStringsSerDe accordingly. Also fixing a 3-byte length-prefix read that corrupted string summaries.

default performance:

Building compact tuple sketch: lg_k=19 (k=524288), n=524288
  build time: 0.27 s, estimate=524288, retained=524288
           serialize: mean=  176.46 ms  stdev=   0.00 ms  min=  176.46 ms  n=1
  serialized size: 6,291,472 bytes
         deserialize: mean=73239.24 ms  stdev=   0.00 ms  min=73239.24 ms  n=1

performance with memory view:

Building compact tuple sketch: lg_k=19 (k=524288), n=524288
  build time: 0.21 s, estimate=524288, retained=524288
           serialize: mean=  147.77 ms  stdev=   0.00 ms  min=  147.77 ms  n=1
  serialized size: 6,291,472 bytes
         deserialize: mean=  150.76 ms  stdev=   0.00 ms  min=  150.76 ms  n=1

bench_tuple_serde.py

@cboumalh

cboumalh commented Nov 10, 2026 •

Copy link
Copy Markdown
Author

Hi @leerho @tisonkun @proost, hope all is well. Please let me know if this is worth pursuing!

@cboumalh

Copy link
Copy Markdown
Author

@leerho @jmalkin @jaykz52
Hi all, just checking in on this PR. It cuts deserialization of a large tuple sketch from ~73s to ~150ms and fixes a string summary bug. Happy to change anything. Thanks!

@tisonkun

Copy link
Copy Markdown
Member

Thanks for your contribution! I will take a review today.

@cboumalh

Copy link
Copy Markdown
Author

no problem @tisonkun, thank you for taking the time! Please let me know if there is anything I can help with.

A Python SerDe can retain a slice of its input as the decoded value. A view created from a raw pointer does not retain the serialized sketch and can outlive its storage.

Carry the original bytes through each sketch reader and build the view from that object. Translate the native cursor to an offset in the image so retained slices stay valid without copying the remaining input for every item.
Existing from_bytes implementations use bytes methods such as decode and index. Pass the original bytes object and the offset of each item directly to the callback so these readers work without copying the remaining input.
Read the four-byte little-endian length written by PyStringsSerDe and check that both the prefix and payload fit from the supplied offset. Python slices silently truncate, so a short payload could otherwise be returned as a complete string.

Exercise a 16 MiB string and truncated records directly. These cases replace the repeated tuple fixtures and their string update policy.

@tisonkun tisonkun left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I pushed three follow-up commits to keep retained summary views valid, preserve the bytes API for custom SerDes, and validate string lengths. I also simplified the tests around those cases.

Looks good to me. Please take another look at the follow-ups when you have a chance.

@cboumalh cboumalh left a comment

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just reviewed, thanks for simplifying this further!

@tisonkun
tisonkun merged commit 74efd2d into apache:main Nov 19, 2026
3 checks passed
@tisonkun

Copy link
Copy Markdown
Member

Thanks for your contribution @cboumalh!

Although, I have no idea when the next release will be scheduled. I would appreciate it if you could share your use case and see whether a recent release can help in your case.

@cboumalh

cboumalh commented Nov 19, 2026 •

Copy link
Copy Markdown
Author

@tisonkun Thanks for the review :)

We're deserializing large tuple sketches at scale through PySpark jobs at Amazon. On the AWS service we use, we don't have any other options besides using the library directly since the service hasn't adopted Spark 4.2 (supports tuple sketches natively), and we can only use PySpark. Currently we have to write the deserializer from scratch to overcome the bottleneck caused by the bug we just fixed. The sooner this gets released the better! If that is possible.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants